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Introduction 


A machine vision algorithm that can generate pointing solutions for the HEROES payload 
based on images from the Pitch and Yaw Aspect System (PYAS) in real time 






Introduction 


• Sun sensing is a common problem 

■ Solar observatory pointing 

■ General spacecraft attitude 

■ Solar panel pointing 

• What makes this a challenging vision problem? 

■ Sub-pixel accuracy is required 

■ Computation time is limited 

• How was this approach unique? 

■ All off-the-shelf electronics 

■ Long focal length 
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HEROES Mission 







Consisted of modifications to an existing hard x- 
ray telescope payload: HERO 

■ HERO used a star camera for pointing 

■ Star camera could not provide pointing 
solutions near or at the sun 

■ PYAS replaced star camera for solar pointing 


HEROES was a high altitude balloon mission for 
making hard x-ray observations of both the sun 
and of astrophysical targets 

Launched from Fort Sumner, NM in Fall 2013 


Aspect 
System 
omputers 
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Pitch and Yaw Aspect 
Systems (PYAS) 


PYAS Requirements 


• The star camera on HEROES 
provided fine pointing solutions 


• During solar observation, the PYAS 
needed to provide solutions with 
similar cadence and accuracy 


• PYAS needed a fully visible sun to 
generate fine solutions, but could 
generate a coarse solution with a 
partially-visible sun 



Cadence 

Accuracy 



1 Hz 

20 arcsec 
(~1.9 pixels) 
2.8° (fine) 
>3.3° (coarse) 


Field of View 




PYAS Overview 

• Each PYAS system is 3 meters long 

• Camera, lens and filters are at one end, fiducial plate at 
the other 

• Optical path is enclosed with a cardboard baffle 

• No moving parts or powered elements other than 
heaters, camera, and computer 







PYAS Optical Path 
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Fiducial Pattern 


• Gives a way to convert from image 
coordinates to gondola coordinates 

• Printed pattern of identical cross-shaped 
marks on a metal plate 

• Identity of each fiducial is encoded in the 
distance between adjacent fiducials 

• Need a minimum of 3 adjacent non- 
collinear fiducials to completely identify 

• Only a small portion of the fiducial plate 
is illuminated in any PYAS image 




PYAS Image 
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Algorithm 


• The PYAS algorithm was broken into four 
basic stages: 

■ Locate Sun 

■ Locate Fiducials 

■ Identify Fiducials 

■ Transform Sun Center 


• The algorithm takes a raw PYAS image 
computes location of the sun relative to 
the center of the fiducial screen 









Assumptions 


• There will only be one sun-like object present in each frame 

• Optics and camera are approximately parallel to the fiducial plate 

■ Solar image is circular rather than elliptical 

■ Projective effects from the camera orientation can be approximated with a similarity 
transform 

■ Clocking of fiducial plate relative to the camera is negligible 

■ Change in distance and clocking between fiducial plate and camera will be small 

• The projected solar image will not be under or overexposed, and required exposure 
settings will not change drastically over the duration of a flight 




Related Work 


• Sun Tracking 

■ RHESSI SAS 

■ Average Intersection for circles 

• Fiducial Detection 

■ Common machine vision problem 

■ Commonly solved with intensity correlation 

■ Sub-pixel location can be determined with a centroid 


true positions of the three CCD’s 
with reconstructed Sun center 



residual tnangle with 
shifted Sun center 


wrong calibrated CCD with 
wrong limb position 
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M. Fivian, R. Henneck, and A. Zehnder, "RHESSI Aspect System and In-flight 
Calibration," Proc. SPIE, vol. 4853, pp. 60-70, 2003. 
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Chord Placement 


• Motivated by RHESSI Solar Aspect 
System 

• Allows for an easy trade between 
computational complexity and accuracy 

• Made use of two grids, coarse and fine 

■ Coarse grid looks for chords evenly 
distributed over entire image 

■ Fine grid looks for chords only 
where the sun is expected to be 






Chords Algorithm 


• Multiple rows and columns are 
inspected in each image 


• Each is compared against a set of 
criteria to assess whether or not it is a 
chord through the sun 


• Valid chords are compared against a 
brightness threshold to determine the 
location of the solar limb 
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Chords Algorithm 

•For each chord, edges of the sun are 
located and refined with a linear fit in 
intensity 


•Edges are averaged to get midpoint of 
the chord 

•The resulting midpoints are averaged 
for each axis to determine the center of 
the sun 

•ROI around this center is used for 
fiducial detection 
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Algorithm 
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Fiducial Detection 


• Fiducials are located using an edge- 
based shape detection mask 

• Correlation with the mask yields an image 
with local maxima at the locations 
corresponding to fiducials 


• Location is refined to sub-pixel level with 
a centroid around correlation maxima 






Algorithm 
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Fiducial Identification 


• Inter-fiducial distances are computed 
along rows and columns of the image 
for every pair of detected fiducials 

• Recall that fiducials are spaced a fixed 
distance apart along rows or columns 
with the same ID 

• The list of distances is searched for 
any that match the fixed spacing 

• For each match, the other axis is 
compared to the list of valid spacings 

• If there is a match, each fiducial in the 
pair is identified in that axis 





Algorithm 
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Mapping 


• Converting from pixel space to gondola coordinates was handled with a pair of linear fits 

• Once fiducials are identified, their locations on the plate could be looked up, giving a 
correspondence pair between the camera and the plate 

• A separate scale and offset was computed for each axis to convert from the camera 
coordinate system to the fiducial plate 


• Calibrations on the ground gave the mapping between the plate and the gondola 
coordinate system used by the HEROES pointing controller 
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Performance Testing 


• Synthetic Data 

■ Circle Finding 

■ Fiducial Detection 

• PYAS Test Data 

■ Fiducial Identification 

■ Mapping 

• Flight Performance 




Circle Finding Tests 


• Could not devise a test setup with a 
known reference 

• Tests were all performed synthetic data 

• Looking to measure 

■ Jitter in solution 

■ Error bias 

■ Trade between chords and accuracy 

■ Effects of noise 

■ Effect of fiducials 





Real Data 


Synthetic Data 





Circle Finding: Number of Chords 


• Interested in how error is affected by the 
number of chords used to generate a 
solution 

• Observed a sharp decline in error as a 
function of chords used 

• Errors were roughly converted from 
pixels to arcseconds with ideal scale 
factor 

• Severely diminishing returns after 10 
chords per axis 


Error vs Number of Chords 
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Circle Finding: Noise Tolerance 


Error vs Noise Level 


• Average Intersection is known to be very 
vulnerable to noise 

• With a fixed number of chords, noise 
level was varied to assess impact on 
precision and valid chords 

• Number of usable chords falls off sharply 
and error begins to climb if noise is 
beyond 15 LSB 

• PYAS test data was measured to have a 
noise level of approximately 10 LSB 




Valid Chords 
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Noise Level (counts) 



Circle Finding: Effect of Fiducials 


• Tests were performed on artificial data 
with a fiducial field added 

• Test data had a fixed center but a wide 
range of fiducial locations 

• A histogram of error magnitude is plotted 
at left 

• Almost all values fall below 1 0 arcsec 


RMS Error 
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Fiducial Detection Tests 


• Effects of error in fiducial location will be 
rolled into testing of mapping 


• More concerned here to see if fiducial 
detection has any systematic error 


• Tests were run on images of a single 
artificially generated fiducial 







Fiducial Detection 


• Fiducial detection showed an average 
error of 1 .55 arcsec on synthetic data 


• Closer inspection shows that systematic 
error was introduced when refining the 
location of the fiducial to sub-pixel levels 


• Centroid was computed on pixels near 
peak which were above a threshold 

• By not subtracting that threshold, points 
on the edge of the neighborhood had 

effect on the center 
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Fiducial Identification 


• Difficult to test in any way other than 
manual inspection 

• Fully illuminated fiducials were correctly 
identified in ground testing 

• Partially illuminated fiducials were often 
not found or identified, but were not often 
falsely identified 

• Scratches tripped false fiducials in 
testing, were removed by adjusting 
brightness of brightest points in the 
image 

• Removed by adjusting outlier brightness 






Mapping Tests 


• Effects of mapping from the camera 
plane to the fiducial plane were 
measured directly on PYAS test data 

• Noise in fiducial locations led to noise in 
mapping coefficients, which in turn 
introduced noise in the pointing solution 


• Applying measured jitter in mapping 
coefficients from real data to jitter in sun 
center from synthetic data, overall jitter 
had a 3 a of 19.5 arcsec 
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Flight Performance 

• HEROES launched on Sep 21 , 2013 

• Flight lasted over 24 hours, with 7 
devoted to solar observation 

• PYAS maintained a cadence of 3.97 Hz 


• Requirement for overall pointing was that 
it be within 60 arcseconds of the target 
50% of the time 

• PYAS showed a a 50 th percentile of 1 0 
arcsec and 30 arcsec in azimuth and 
elevation respectively 
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S. Christe, A. Shih, et. al, "A Solar Aspect System for the HEROES Mission," IEEE 
Aerosp. Conf., 2014. 
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Lessons Learned 


• Basing algorithm on RHESSI seemed like a good at the outset, in hindsight a more 
conventional approach would likely have been better 


• Algorithm would benefit from a complete restructuring, locating fiducials before 
searching for sun center 


• Circular fiducials may have been easier to detect, a more complicated fiducial pattern 
might have been easier to identify 


• Improvements to computer hardware, image storage, or smarter camera interactions 
could allow for higher cadence and more complex algorithms 



Questions? 




